Repository navigation
Conversation
…8TensorStats (NVIDIA#2801) - Register scale_inv_std (plus helper variance/numel/sum buffers using Welford reduction) for all FP8 recipes and NVFP4 in add_scale_inv_stats. Population variance keeps std=0 for delayed/current scaling where scale_inv is a single scalar. - Also wire scale_inv_min/max/std for NVFP4 (was previously only FP8 recipes). - LogFp8TensorStats.inspect_tensor now filters bare stats on NVFP4 layers with a warning instead of raising, so dual LogFp8TensorStats + LogNvfp4TensorStats configs work with overlapping (or catch-all) layer regexes. Recipe-prefixed FP8 stats (e.g. mxfp8_mse) are preserved for what-if comparisons. - Numerics test extended to validate scale_inv_min/max/std against torch.std(scale_inv, unbiased=False). Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
for more information, see https://pre-commit.ci
Collaborator
Author
|
/te-ci pytorch |
Contributor
|
Parallel-group variance is Sigma n_i*(var_i + (mean_i - mean)^2) / N - the between-group term must be added, not subtracted. Single-group buffers hide the bug (mean_i = mean_global so the term is 0); it surfaces with scale_inv_std reduced across microbatches/ranks, where negative variance flows into sqrt() and yields NaN. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
ptrendx
reviewed
Jun 1, 2026
|
|
||
| def check_if_stat_is_supported(self, stat: str): | ||
| """Returns True if stat is supported, raises ValueError otherwise.""" | ||
| bare = stat[: -len("_columnwise")] if stat.endswith("_columnwise") else stat |
Member
There was a problem hiding this comment.
mse_columnwise and underflow%_columnwise should not pass this function but they would with the logic as written.
Collaborator
Author
There was a problem hiding this comment.
I added them to exception, even though we should have stats for them I guess, but this is not scope of the PR.
…validation compute_variance/compute_std now combine on M2 with an explicit `unbiased` flag. Previously sample variances (torch.var default, used by variance/std) were fed into a population-style combine, so reductions across microbatches/ranks were silently wrong whenever group means differed. scale_inv_* combinators now pass unbiased=False to match their population feeders. Adds test_stats_computation_microbatch_reduction which sweeps the stats registry and checks every aux-free combinator against its own compute fn over the concatenation. NVFP4 check_if_stat_is_supported no longer accepts mse_columnwise / underflows%_columnwise: only scale_inv_* have a columnwise variant, the others are computed from the single quantized tensor and would crash downstream. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> # Conflicts: # transformer_engine/debug/features/utils/stats_computation.py
…tyle Collapse import/list/call reflows that were split by an accidental black run at the default line length (88) instead of the repo's 100; no functional change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…d reduction test Route every scale_inv stat (min/max/std/variance/numel/sum) through a shared real_scale_inv() helper that strips the zero padding MXFP8/NVFP4 quantizers add, so padding no longer deflates the mean or corrupts the numel-weighted variance reduction across microbatches/ranks. Add a parametrized test driving the aux-dependent parallel-variance combine for scale_inv_std and checking it against std over the concatenated scale_inv values. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
for more information, see https://pre-commit.ci
…unbiased convention Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…ch reduction test Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Collaborator
Author
|
/te-ci pytorch L1 |
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Ignore zero scale entries in numerical references consistently with the logger. Cover all recipes, both scale orientations, padding, and combined FP8/NVFP4 logging. Report unsupported NVFP4 stats with ValueError. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Addresses items 1 and 2 of #2801:
scale_inv_stdfor delayed/current FP8 scaling, MXFP8, FP8 block scaling and NVFP4, including supported_columnwisevariants. NVFP4 also gainsscale_inv_minandscale_inv_max.LogFp8TensorStatsandLogNvfp4TensorStatsto share a layer selection. On NVFP4 layers, the FP8 logger skips bare stats with a warning and preserves FP8-recipe-prefixed what-if stats such asmxfp8_mse.Scale standard deviation uses population variance, so a single scale has
std=0. Microbatch/rank reduction uses the Chan/Welford combination. This also fixes the existing between-group sign error and preserves the sample-variance convention of the high-precisionvariance/stdstatistics.Per-block metadata dumping (item 3 of #2801) remains outside this PR.
Validation
gb102):tests/pytorch/debug/test_log.py— 31 passed, 8 skipped. Tested the changed Python features with TE 2.20.0.dev0 / PyTorch 2.14.0a0 native binaries from the current TE devel image.PYTHON_ONLY=1 qa/L0_pytorch_lint/test.sh: passed. Black 24.4.2: passed.Type of change